Gesture onset detection on multiple devices

ABSTRACT

Implementations of the disclosed subject matter provide techniques for improved identification of a gesture based on data obtained from multiple devices. A method may include receiving an indication of an onset of a gesture, from a first device, at a gesture coordinating device. Next, first subsequent data describing the gesture may be received from a second device, at the gesture coordinating device. Based on the indication and the first subsequent data, the gesture may be identified. In response to identification of the gesture, an action may be performed based on the gesture identified. In some cases, the gesture coordinating device may be a cloud-based device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.14/218,168 filed Mar. 18, 2014, the contents of which are incorporatedby reference.

BACKGROUND

Accurate detection of a gesture based on the motion of a device (e.g.,handheld or wearable) can be difficult to implement. Typically,unreliable sensors such as accelerometers and gyroscopes may be used.However, only a limited amount of information describing a gesture canbe discerned on a single device using these unreliable sensors. In somecases, a particular gesture may require some means of verifying thedirection of a gesturing device relative to another device (e.g.,another device to which the user wishes to transfer content) and toidentify the other device toward which a gesture is being aimed. Thistype of gesture can be difficult to detect using only one on-devicesensor. For example, it's impossible to know which direction a device ispointing using unreliable sensors such as accelerometers and gyroscopes.

BRIEF SUMMARY

According to an embodiment of the disclosed subject matter a method mayinclude receiving an indication of an onset of a gesture, from a firstdevice, at a gesture coordinating device. Next, first subsequent datadescribing the gesture may be received from a second device, at thegesture coordinating device. Based on the indication and the firstsubsequent data, the gesture may be identified.

An implementation of the disclosed subject matter provides a systemincluding a processor configured to receive an indication of an onset ofa gesture, from a first device, at a gesture coordinating device. As aresult, first subsequent data describing the gesture may be receivedfrom a second device, at the gesture coordinating device. Based on theindication and the subsequent data, the gesture may be identified.

In an implementation, a system according to the disclosed subject matterincludes means for receiving an indication of an onset of a gesture,from a first device, at a gesture coordinating device. The systemfurther includes means for receiving first subsequent data describingthe gesture, from a second device, at the gesture coordinating deviceand identifying the gesture based on the indication and the firstsubsequent data.

Implementations of the disclosed subject matter provide techniques forimproved identification of a gesture based on data obtained frommultiple devices. By combining data obtained from a first device and oneor more other devices, identification of a gesture may be more accurateand the direction of a gesture may be verified. Additional features,advantages, and embodiments of the disclosed subject matter may be setforth or apparent from consideration of the following detaileddescription, drawings, and claims. Moreover, it is to be understood thatboth the foregoing summary and the following detailed description areexamples and are intended to provide further explanation withoutlimiting the scope of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the disclosed subject matter, are incorporated in andconstitute a part of this specification. The drawings also illustrateembodiments of the disclosed subject matter and together with thedetailed description serve to explain the principles of embodiments ofthe disclosed subject matter. No attempt is made to show structuraldetails in more detail than may be necessary for a fundamentalunderstanding of the disclosed subject matter and various ways in whichit may be practiced.

FIG. 1 shows an example process according to an implementation of thedisclosed subject matter.

FIG. 2 shows an example system arrangement according to animplementation of the disclosed subject matter.

FIG. 3 shows an example information flow according to an implementationof the disclosed subject matter.

FIG. 4 shows a computer according to an embodiment of the disclosedsubject matter.

FIG. 5 shows a network configuration according to an embodiment of thedisclosed subject matter.

DETAILED DESCRIPTION

Detection of a gesture based on the motion of a device (e.g., handheldor wearable) can be difficult to implement, inaccurate, and falselytriggered. Typically, unreliable sensors such as accelerometers andgyroscopes may be used to detect the occurrence of a gesture on adevice. However, a limited amount of information may be discerned on asingle device using these unreliable sensors. For example, it'simpossible to know which direction a device is pointing using unreliablesensors such as accelerometers and gyroscopes.

Increasingly there are other sensors in the environment, such as camerasand microphones. These can be found on traditional devices like TVs, butalso on smart devices/appliances. These sensors can be used to increasethe accuracy of detecting a gesture, and also facilitate directionalityrelative to other devices and/or landmarks in a room. However, using allof the sensors in the environment may require cameras that arepermanently recording and microphones that are permanently receivingaudio signals. Requiring sensors to constantly remain on to continuouslyreceive gesture data can have privacy and power consumptionimplications.

The present disclosure provides a multi-step gesture detection techniquethat includes increasingly sophisticated gesture detection usingmultiple sensors on multiple devices. As an example, a gesture onsetdetector (D1) may rely on one or more sensors (S1) that may continuouslyrun on a first device. D1 may be a power-efficient gesture onsetdetector and may run continuously without significant effect on thedevice. As a user of the first device begins to perform a gesture on thefirst device, the gesture onset detector (D1) may be triggered. As aresult, the onset of the gesture may be detected by the gesture onsetdetector (D1). The first device may provide, to a gesture coordinatingdevice, an indication of the onset of the gesture. To verify the user'sintent and the occurrence of the gesture, another gesture detector (D2),on a second device, may be triggered by an instruction received from thethird coordinating device or the first device. Gesture detector (D2) maybe turned on and may rely on one or more sensors (S2), and in somecases, the one or more sensors (S2) may be of a different type of sensorthan the one or more sensors (S1). For example, S1 may be an inertialmeasurement unit (IMU) and S2 may be a camera. If the gesture detectorD2 receives data from the one or more sensors S2 indicating theoccurrence of the gesture, there may be an increased likelihood of theuser's intent and the gesture may be identified. Additional gesturedetectors using one or more other sensors (e.g., D3 and sensor S3, D4and sensor S4 . . . Dn and Sn), on any combination of the first device,the second device, and/or one or more other devices, may also be used toidentify the gesture. In general, gesture detectors can span multipledevices and a set of sensors can span multiple devices as well.Additionally, an arbitrary number of devices may be used to identify agesture. Once a gesture has been identified based on a predefined numberof gesture detectors and associated sensors, the gesture detectors (D2,D3, etc.) may be turned off and their respective sensors (S2, S3, etc.)may be disabled. In some cases, the gesture onset detector D1 and theone or more sensors S1 associated with D1 may continuously run on thefirst device.

A feature of the presently disclosure subject matter is that rather thancontinuously running costly sophisticated sensors such as cameras andmicrophones on one or more devices, economical simple sensors to detectthe onset of a gesture that is likely to be occurring, and thensubsequently instructing the costly sophisticated sensors to turn on, onone or more other devices, temporarily in order to improve quality ofgesture identification. An advantage of the techniques disclosed hereinis that costly sophisticated sensors such as cameras and microphones maybe turned on less often, thereby reducing the likelihood that potentialattackers may access fragmentary streams of information from thesesensors. In addition, since costly sophisticated sensors may only beactivated as needed, rather than remaining turned on all the time, powermay be saved across one or more various devices.

Various implementations of the disclosed subject matter are provided.One option may be to use a gesture onset detector on a device to triggeradditional functionality on the device itself. For example, anaccelerometer and/or gyroscope may obtain motion data on the device suchas a smartphone, which may trigger the camera on the smartphone to takea photo in order to verify the direction the phone is aiming. In thiscase, the motion data and the direction of the phone may be used toidentify a gesture. Another option may be to use a gesture onsetdetector on a first device to trigger additional functionality on asecond device. For example, an accelerometer and/or gyroscope may obtainmotion data on the first device, which may trigger the camera on asecond device to take a photo in order to verify the direction the firstdevice is aiming. In this case, the motion data from the first deviceand the direction of the first device obtained by the second device maybe used to identify a gesture. Yet another option may be to use thegesture onset detector on a first device to trigger additionalfunctionality on both the first device and a second device. For example,an accelerometer and/or gyroscope may obtain motion data on the firstdevice, which may trigger the microphone on the first device to turn onand emit an audio signal. At the same time, the speaker on the seconddevice may be triggered to turn on in order to verify that first devicemoved toward the second device based on Doppler shift. In this case, themotion data from the first device and the direction of the first deviceobtained by the Doppler shift data between the first and second devicesmay be used to identify a gesture. Alternatively, or in addition,audio-based features other than Doppler shift may be used. For example,relative amplitude may be used in cases in which there are multipleemitting speakers, and time-of-flight audio techniques may beimplemented.

In an implementation in which data obtained from a gesture onsetdetector on a first device is combined with additional data obtainedfrom one or more sensors on one or more other devices, a gesturecoordinating device may be used. A gesture coordinating device mayfacilitate communication between a first device and one or more otherdevices across one or more bridges between the interfaces. Specifically,the gesture coordinating device may receive an indication of the onsetof a gesture from a first device and subsequent data describing thegesture from one or more other devices. As a result, the gesturecoordinating device may identify the gesture based on the indicationreceived from the first device and the subsequent data received from theone or more other devices. According to an implementation, a gesturecoordinating device may be a cloud-based device.

In general, implementations of the disclosed subject matter provide asystem that includes a gesture onset detector on a device that maytrigger a cascade of more precise detectors on one or more otherdevices. The present disclosure provides techniques for improvedidentification of a gesture based on data obtained from multipledevices. FIG. 1 shows an example process according to an implementationof the disclosed subject matter. As shown, a method may includereceiving an indication of an onset of a gesture, from a first device,at a gesture coordinating device, at 101. Next, first subsequent datadescribing the gesture may be received from a second device, at thegesture coordinating device, at 102. A device as used herein may be atablet, a wearable computer, a smartphone, a laptop, a mobile device, aset-top box, a PC, a TV, an audio system, and another other device whichmay be used to detect a gesture. A device may include one or moresensors such as an ambient light sensor, an accelerometer, a gyroscope,a camera, a magnetometer, a speaker, a microphone, a touch screen, andthe like. For example, the onset of the gesture may be detected by oneor more sensors on the first device and a corresponding indication maybe provided to the gesture coordinating device. Similarly, one or moresensors on a second device may be used to obtain additional datadescribing the gesture, the onset of which was detected by the firstdevice. The second device may provide this subsequent data describingthe gesture to the gesture coordinating device. In some cases, followingreceipt of an indication of the onset of a gesture from a first device,the first device and/or the gesture coordinating device may instruct thesecond device to activate a second sensor that may capture the firstsubsequent data. Based on the indication of the onset of the gesture,received from the first device, and the first subsequent data describingthe gesture, received from the second device, the gesture may beidentified, for example, by the gesture coordinating device, at 103.

FIG. 2 shows an example system arrangement according to animplementation of the disclosed subject matter. As shown, a user may beusing a device 201, such as a smartphone. The user may wish to transfercontent, currently being displayed on smartphone 201, to a device 202,such as a TV, using a gesture performed on the device 201 towards thedevice 202. A gesture may be any interaction by a user with a device,such as moving, waving, shaking, pointing, tapping, touching, and thelike, of a device. The user may perform a gesture motion by moving thesmartphone 201 toward TV 202. Smartphone 201 may detect the onset of thegesture based data obtained by one or more sensors on the device. Forexample, motion data 203 may be obtained from an IMU of the smartphone201. As a specific example, motion data 203 may include an angleassociated with the yaw (e.g., α₁) of the smartphone 201, an angleassociated with the pitch (e.g., β₁) of the smartphone 201, an angleassociated with the roll (e.g., γ₁) of the smartphone, and/or anacceleration of the smartphone. Although not shown in FIG. 2, the motiondata may include a stream of readings from an IMU such as anaccelerometer on the smartphone 201. In this case, detection of theonset of the gesture may not depend on the static state of thesmartphone 201, but rather on a motion signature over time. Thesmartphone 201 may provide an indication of the onset of the gesture toa gesture coordinating device (not shown). The smartphone 201 and/or thegesture coordinating device may instruct the TV 202 to activate a sensor204 on the TV 202 to obtain subsequent data describing the gesture. Insome cases, the detection of the onset of the gesture from smartphone201 may trigger the TV 202 to activate sensor 204, such as a camera. Thecamera 204 may be used obtain subsequent data describing the gesture,for example, the camera 204 may be used to determine that the smartphone201 is pointed in a direction towards the TV 202. The gesturecoordinating device may receive this subsequent data from the TV 202.Based on the motion data 203 and the received subsequent data from TV202 using camera 204, the gesture coordinating device may identify thegesture. For example, gesture coordinating device may identify thegesture as intending for content to be transferred to the TV 202. Insome cases, the gesture coordinating device may provide an indication ofthe identified gesture to the first device, the second device, and/orany other device associated with the identified gesture.

As a result of identifying the gesture, an action may be performed basedon the gesture identified. An action may be performed by the firstdevice, the second device, the gesture coordinating device, and/or anyother device that may receive an indication of the identified gesture.As in the example above, based on the identified gesture, e.g., that theuser intends to transfer content from the smartphone to the TV, thecontent may be transferred to the TV 202. For example, the content maybe transferred to the TV 202 from either the smartphone 201, the gesturecoordinating device, or from a remote server.

In some cases, identification of a gesture by a gesture coordinatingdevice may be based on data and/or information in addition to a receivedindication of the onset of a gesture and subsequent data. As in theexample above, a gesture by a user of a smartphone intending to transfercontent to a TV may implicate security issues if the gesture is falselyidentified. For example, if a gesture is falsely identified, content maybe transferred to a device when the user may not have intended to causesuch a transfer. Accordingly, additional data and/or information may beused to identify a gesture by a gesture coordinating device to avoidfalsely triggering an action based on a falsely identified gesture.Referring back to FIG. 2, according to an implementation, a first tokenmay be received, at the gesture coordinating device, from the firstdevice in addition to the indication of the onset of a gesture.Similarly, a second token may be received, from the second device, inaddition to the subsequent data describing the gesture. The gesturecoordinating device may activate an authentication protocol to determinethat the user intends the gesture between the two devices identified bythe first and second tokens. For example, the gesture coordinatingdevice may also use GPS location information from each device to confirmthat the devices are co-located. In this case, identification of thegesture by the gesture coordinating device may be further based on anauthentication of the devices and verification of the user's intentbased on the gesture. For example, if the gesture coordinating device isunable to authenticate the devices or does not receive a token from oneof the devices at all, the gesture coordinating device may not identifythe gesture.

As mentioned above, the detection of the onset of a gesture on a firstdevice may trigger additional functionality on both the first deviceand/or a second device. Referring back to FIG. 2, an accelerometerand/or gyroscope may obtain motion data 203 on the first device 201,which may trigger the camera 204 on the second device 202 to turn on andobtain first subsequent data, such as an image depicting the directionof the first device 201 relative to the second device 202. In addition,the gesture coordinating device may receive second subsequent data fromthe first device, which may be triggered by the onset of the gesturedetected at the first device. As an example, the detection of the onsetof a gesture on the first device 201 may also trigger a camera (notshown) on the first device 201 to turn on and obtain second subsequentdata, such as another image depicting the direction of the first 201relative to the second device 202. In some cases, the gesturecoordinating device may instruct the first device to activate a thirdsensor that captures the second subsequent data. In this case, themotion data 203 from the first device 201, the first subsequent datareceived from the second device 202 and the second subsequent datareceived from the first device 201 may be used to identify a gesture.

According to an implementation, identifying the gesture based on theindication of the onset of the gesture and the first subsequent data mayinclude determining that each of the indication and the first subsequentdata exceeds a gesture detection threshold. For example, a gesturecoordinating device may receive an indication of the onset of a gesturewhich may include data such as motion data 203. Motion data 203 mayinclude an angle associated with the yaw (e.g., α₁) of the smartphone201, an angle associated with the pitch (e.g., β₁) of the smartphone201, an angle associated with the roll (e.g., γ₁) of the smartphone,and/or an acceleration of the smartphone. The gesture coordinatingdevice may evaluate the motion data 203 to determine if each or acombination of the data (α₁, β₁, γ₁) and/or the acceleration of thefirst device 201 exceeds one or more thresholds associated with the datain order to identify the gesture. The gesture coordinating device maysimilarly evaluate the subsequent data received from the second device,the first device, and any other device from which subsequent data may bereceived. For example, if the acceleration of the first device 201 doesnot exceed an acceleration threshold X and/or the directional datareceived from the second device 202 does not exceed a directionalthreshold Y, the gesture coordinating device may not identify thegesture. In some cases, a threshold for identifying a gesture may bebased on the time period during which the indication of the onset of agesture and subsequent data from one or more devices is received by thegesture coordinating device. For example, the gesture coordinatingdevice may evaluate the data received from multiple sensors on multipledevices with an integrated highly accurate master timing clock. Bysimultaneously evaluating the data received from multiple sensorscombined with timing information, from a synchronized clock, associatedwith the data from each of the multiple sensors, a gesture may beidentified using sensor fusion. This may result in more accurate gestureidentification by the gesture coordinating device.

In some cases, a gesture detection threshold may be based on the numberof sensors, the types of data, and/or devices from which subsequent datadescribing the gesture is received by the gesture coordinating device.For example, the gesture coordinating device may identify a gesture ifdata is received from a minimum of X sensors, located on one or moredevices. Similarly, the gesture coordinating device may identify thegesture if data is received from a minimum of Y devices. As anotherexample, the gesture coordinating device may identify a gesture if apredefined combination of types of data are received such as motiondata, angle data, directional data, image data, Doppler shift data, andthe like. Any other threshold, data, and/or information may be used toidentify the gesture.

FIG. 3 shows an example information flow according to an implementationof the disclosed subject matter. As shown, a first device may detect theonset of a gesture, at 301. A gesture coordinating device may receive anindication of the onset of the gesture from the first device, at 302. Insome cases, in response to the detection of the onset of the gesture bythe first device, the second device may obtain subsequent datadescribing the gesture, at 303. Next, the gesture coordinating devicemay receive the subsequent data from the second device, at 304. Based onthe indication of the onset of the gesture and the subsequent data, thegesture coordinating device may identify the gesture, at 305. Althoughnot shown in FIG. 3, the first device and the second device may be indirect communication with one another. For example, the first device mayindicate, to the second device, the detection of the onset of thegesture at the first device which may trigger a sensor to be activatedon the second device to obtain subsequent data describing the gesture.Similarly, the second device may provide subsequent data describing thegesture to the first device. In this case, the first device may identifythe gesture and provide an indication of the identified gesture to thegesture coordinating device. Alternatively, the first device maysubsequently provide the subsequent data to the gesture coordinatingdevice which may identify the gesture. As another example, the firstdevice may provide an indication of the identified gesture to thegesture coordinating device, and the third coordinating device mayperform an action and/or provide an instruction to another device basedon the gesture identified. In some cases, multiple coordinating devicesmay be used, for example, in an implementation in which multiple devicesare used to identify a gesture.

In some cases, the gesture coordinating device and/or the first devicemay trigger, using an audio signal, a sensor to be activated on thesecond device to obtain subsequent data describing the gesture. Becausea sensor on the second device may need to be activated quickly as thegesture is occurring, one technique may be to use an audio signal to actas this trigger. In this case, all devices in an environment (e.g., aroom) may be constantly listening for an audio signal such as by using amicrophone, which has much lower power requirements than say a camera ora gyroscope, that may remain turned on. When the gesture coordinatingdevice or the first device detects the onset of a gesture, the gesturecoordinating device or the first device may immediately emit apre-determined ultrasonic audio sequence at a volume level that is highenough for at least one of the devices in the environment to receive theaudio sequence. As a result, immediately upon receiving the audiosequence, the one or more other devices, may automatically activate oneor more additional sensors on the device to obtain subsequent datadescribing the gesture, thereby significantly reducing any lag orlatency in identifying a gesture.

Embodiments of the presently disclosed subject matter may be implementedin and used with a variety of component and network architectures. FIG.4 is an example computer system 20 suitable for implementing embodimentsof the presently disclosed subject matter. The computer 20 includes abus 21 which interconnects major components of the computer 20, such asone or more processors 24, memory 27 such as RAM, ROM, flash RAM, or thelike, an input/output controller 28, and fixed storage 23 such as a harddrive, flash storage, SAN device, or the like. It will be understoodthat other components may or may not be included, such as a user displaysuch as a display screen via a display adapter, user input interfacessuch as controllers and associated user input devices such as akeyboard, mouse, touchscreen, or the like, and other components known inthe art to use in or in conjunction with general-purpose computingsystems.

The bus 21 allows data communication between the central processor 24and the memory 27. The RAM is generally the main memory into which theoperating system and application programs are loaded. The ROM or flashmemory can contain, among other code, the Basic Input-Output system(BIOS) which controls basic hardware operation such as the interactionwith peripheral components. Applications resident with the computer 20are generally stored on and accessed via a computer readable medium,such as the fixed storage 23 and/or the memory 27, an optical drive,external storage mechanism, or the like.

Each component shown may be integral with the computer 20 or may beseparate and accessed through other interfaces. Other interfaces, suchas a network interface 29, may provide a connection to remote systemsand devices via a telephone link, wired or wireless local- or wide-areanetwork connection, proprietary network connections, or the like. Forexample, the network interface 29 may allow the computer to communicatewith other computers via one or more local, wide-area, or othernetworks, as shown in FIG. 5.

Many other devices or components (not shown) may be connected in asimilar manner, such as document scanners, digital cameras, auxiliary,supplemental, or backup systems, or the like. Conversely, all of thecomponents shown in FIG. 4 need not be present to practice the presentdisclosure. The components can be interconnected in different ways fromthat shown. The operation of a computer such as that shown in FIG. 4 isreadily known in the art and is not discussed in detail in thisapplication. Code to implement the present disclosure can be stored incomputer-readable storage media such as one or more of the memory 27,fixed storage 23, remote storage locations, or any other storagemechanism known in the art.

FIG. 5 shows an example arrangement according to an embodiment of thedisclosed subject matter. One or more clients 10, 11, such as localcomputers, smart phones, tablet computing devices, remote services, andthe like may connect to other devices via one or more networks 7. Thenetwork may be a local network, wide-area network, the Internet, or anyother suitable communication network or networks, and may be implementedon any suitable platform including wired and/or wireless networks. Theclients 10, 11 may communicate with one or more computer systems, suchas processing units 14, databases 15, and user interface systems 13. Insome cases, clients 10, 11 may communicate with a user interface system13, which may provide access to one or more other systems such as adatabase 15, a processing unit 14, or the like. For example, the userinterface 13 may be a user-accessible web page that provides data fromone or more other computer systems. The user interface 13 may providedifferent interfaces to different clients, such as where ahuman-readable web page is provided to web browser clients 10, and acomputer-readable API or other interface is provided to remote serviceclients 11. The user interface 13, database 15, and processing units 14may be part of an integral system, or may include multiple computersystems communicating via a private network, the Internet, or any othersuitable network. Processing units 14 may be, for example, part of adistributed system such as a cloud-based computing system, searchengine, content delivery system, or the like, which may also include orcommunicate with a database 15 and/or user interface 13. In somearrangements, an analysis system 5 may provide back-end processing, suchas where stored or acquired data is pre-processed by the analysis system5 before delivery to the processing unit 14, database 15, and/or userinterface 13. For example, a machine learning system 5 may providevarious prediction models, data analysis, or the like to one or moreother systems 13, 14, 15.

More generally, various embodiments of the presently disclosed subjectmatter may include or be embodied in the form of computer-implementedprocesses and apparatuses for practicing those processes. Embodimentsalso may be embodied in the form of a computer program product havingcomputer program code containing instructions embodied in non-transitoryand/or tangible media, such as CD-ROMs, DVDs, hard drives, USB(universal serial bus) drives, flash drives, or any other non-transitorymachine readable storage medium, such that when the computer programcode is loaded into and executed by a computer, the computer becomes anapparatus for practicing embodiments of the disclosed subject matter.Embodiments also may be embodied in the form of computer program code,for example, whether stored in a non-transitory storage medium, loadedinto and/or executed by a computer. When the computer program code isloaded into and executed by a computer, the computer becomes anapparatus for practicing embodiments of the disclosed subject matter.When implemented on a general-purpose microprocessor, the computerprogram code segments configure the microprocessor to create specificlogic circuits. In some configurations, a set of computer-readableinstructions stored on a computer-readable storage medium may beimplemented by a general-purpose processor, which may transform thegeneral-purpose processor or a device containing the general-purposeprocessor into a special-purpose device configured to implement or carryout the instructions. Embodiments may be implemented using hardware thatmay include a processor, such as a general purpose microprocessor and/oran Application Specific Integrated Circuit (ASIC) that embodies all orpart of the techniques according to embodiments of the disclosed subjectmatter in hardware and/or firmware. The processor may be coupled tomemory, such as RAM, ROM, flash memory, a hard disk or any other devicecapable of storing electronic information, as previously described. Thememory or other storage medium may store instructions adapted to beexecuted by the processor to perform the techniques according toembodiments of the disclosed subject matter.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit embodiments of the disclosed subject matter to the precise formsdisclosed. Many modifications and variations are possible in view of theabove teachings. The embodiments were chosen and described in order toexplain the principles of embodiments of the disclosed subject matterand their practical applications, to thereby enable others skilled inthe art to utilize those embodiments as well as various embodiments withvarious modifications as may be suited to the particular usecontemplated.

The invention claimed is:
 1. A computer-implemented method comprising:after a mobile device has begun moving and before the movement of themobile device is determined to be likely associated with a performanceof a particular gesture: obtaining first sensor data that is generatedby a first hardware sensor on the mobile device, transmitting at least aportion of the first sensor data, in response to transmitting at least aportion of the first sensor data, receiving an initial message, inresponse to receiving the initial message, obtaining second sensor datathat is generated by a second hardware sensor on the mobile device,transmitting at least a portion of the second sensor data, and inresponse to transmitting at least a portion of the second sensor data,receiving a subsequent message; and in response to receiving thesubsequent message, performing, by the mobile device, one or moreactions associated with the particular gesture.
 2. Thecomputer-implemented method of claim 1, wherein obtaining first sensordata that is generated by the first hardware sensor on the mobile deviceincludes obtaining motion data using an accelerometer or a gyroscope. 3.The computer-implemented method of claim 2, wherein the motion dataincludes one or more of (i) an angle associated with a yaw of the mobiledevice, (ii) an angle associated with a pitch of the mobile device,(iii) an angle associated with a roll of the mobile device, or (iv) anacceleration of the mobile device.
 4. The computer-implemented method ofclaim 1, wherein obtaining second sensor data that is generated by asecond hardware sensor on the mobile device includes obtaining imagedata using a camera.
 5. The computer-implemented method of claim 1,wherein the initial message instructs the mobile device to (i) activatea camera on the mobile device and (ii) use the camera on the mobiledevice to capture an image.
 6. The computer-implemented method of claim1, wherein obtaining first sensor data that is generated by a firsthardware sensor on the mobile device includes obtaining data that isindicative of the onset of a gesture.
 7. The computer-implemented methodof claim 6, further comprising: in response to obtaining data that isindicative of the onset of a gesture, outputting one or more audiosignals that are used to activate a third hardware sensor on a seconddevice.
 8. A system comprising: one or more computers and one or morestorage devices storing instructions that are operable, when executed bythe one or more computers, to cause the one or more computers to performoperations comprising: after a mobile device has begun moving and beforethe movement of the mobile device is determined to be likely associatedwith a performance of a particular gesture: obtaining first sensor datathat is generated by a first hardware sensor on the mobile device,transmitting at least a portion of the first sensor data, in response totransmitting at least a portion of the first sensor data, receiving aninitial message, in response to receiving the initial message, obtainingsecond sensor data that is generated by a second hardware sensor on themobile device, transmitting at least a portion of the second sensordata, and in response to transmitting at least a portion of the secondsensor data, receiving a subsequent message; and in response toreceiving the subsequent message, performing, by the mobile device, oneor more actions associated with the particular gesture.
 9. The system ofclaim 8, wherein obtaining first sensor data that is generated by thefirst hardware sensor on the mobile device includes obtaining motiondata using an accelerometer or a gyroscope.
 10. The system of claim 9,wherein the motion data includes one or more of (i) an angle associatedwith a yaw of the mobile device, (ii) an angle associated with a pitchof the mobile device, (iii) an angle associated with a roll of themobile device, or (iv) an acceleration of the mobile device.
 11. Thesystem of claim 8, wherein obtaining second sensor data that isgenerated by a second hardware sensor on the mobile device includesobtaining image data using a camera.
 12. The system of claim 8, whereinthe initial message instructs the mobile device to (i) activate a cameraon the mobile device and (ii) use the camera on the mobile device tocapture an image.
 13. The system of claim 8, wherein obtaining firstsensor data that is generated by a first hardware sensor on the mobiledevice includes obtaining data that is indicative of the onset of agesture, and in response to obtaining data that is indicative of theonset of a gesture, outputting one or more audio signals that are usedto activate a third hardware sensor on a second device.
 14. Anon-transitory computer-readable medium storing software comprisinginstructions executable by one or more computers which, upon suchexecution, cause the one or more computers to perform operationscomprising: after a mobile device has begun moving and before themovement of the mobile device is determined to be likely associated witha performance of a particular gesture: obtaining first sensor data thatis generated by a first hardware sensor on the mobile device,transmitting at least a portion of the first sensor data, in response totransmitting at least a portion of the first sensor data, receiving aninitial message, in response to receiving the initial message, obtainingsecond sensor data that is generated by a second hardware sensor on themobile device, transmitting at least a portion of the second sensordata, and in response to transmitting at least a portion of the secondsensor data, receiving a subsequent message; and in response toreceiving the subsequent message, performing, by the mobile device, oneor more actions associated with the particular gesture.
 15. Thecomputer-readable medium of claim 14, wherein obtaining first sensordata that is generated by the first hardware sensor on the mobile deviceincludes obtaining motion data using an accelerometer or a gyroscope.16. The computer-readable medium of claim 15, wherein the motion dataincludes one or more of (i) an angle associated with a yaw of the mobiledevice, (ii) an angle associated with a pitch of the mobile device,(iii) an angle associated with a roll of the mobile device, or (iv) anacceleration of the mobile device.
 17. The computer-readable medium ofclaim 14, wherein obtaining second sensor data that is generated by asecond hardware sensor on the mobile device includes obtaining imagedata using a camera.
 18. The computer-readable medium of claim 14,wherein the initial message instructs the mobile device to (i) activatea camera on the mobile device and (ii) use the camera on the mobiledevice to capture an image.
 19. The computer-readable medium of claim14, wherein obtaining first sensor data that is generated by a firsthardware sensor on the mobile device includes obtaining data that isindicative of the onset of a gesture.
 20. The computer-readable mediumof claim 19, further comprising: in response to obtaining data that isindicative of the onset of a gesture, outputting one or more audiosignals that are used to activate a third hardware sensor on a seconddevice.